A New Probabilistic Model for Recognizing Signs with Systematic Modulations

This paper addresses an aspect of sign language (SL) recognition that has largely been overlooked in previous work and yet is integral to signed communication. This work is the most comprehensive to-date on the recognition of the complex variations in sign appearances due to grammatical processes (inflections). These processes systematically modulate both the temporal and spatial dimensions of a root sign word to convey information in addition to lexical meaning. We propose a novel dynamic Bayesian network – the Multichannel Hierarchical Hidden Markov Model (MH-HMM)– as a modelling and recognition framework for continuously signed sentences that include modulated signs. The MHHMM models the hierarchical, sequential and parallel organization in signing while requiring synchronization between parallel data streams at sign boundaries. Experimental results using particle filtering for decoding demonstrate the feasibility of using the MH-HMM for recognizing inflected signs in continuous sentences.